Head-mount type display device, control system, method of controlling head-mount type display device, and computer program

ABSTRACT

A transmissive head-mount type display device including a display configured to display a virtual image, and capable of transmitting an external sight. The head-mounted display including at least one processor or circuit configured to obtain an object located in a predetermined distance range from the display, and a position of a specific object included in the external sight, control the display to display a virtual image associated with the obtained object, identify a change in the position of the specific object based on a position of the specific object obtained, select the object based on a relationship between the change in the position of the specific object identified and a position of the object obtained, and control the display to display a specific check image associated with the object selected as the virtual image.

This is a Continuation of U.S. application Ser. No. 14/682,409 filedApr. 9, 2015, which claims the benefit of priority of Japanese PatentApplication Nos. 2014-094613 and 2015-024971 filed May 1, 2014 and Feb.12, 2015, respectively. The disclosure of the prior applications arehereby incorporated by reference herein in their entirety.

BACKGROUND 1. Technical Field

The present invention relates to a technology of head-mount type displaydevices.

2. Related Art

A head-mount type display device (a head mounted display (HMD)) as adisplay device to be mounted on the head has been known. The head-mounttype display device generates image light representing an image using,for example, a liquid crystal display and a light source, and thenguides the image light thus generated to the eyes of the user using aprojection optical system and a light guide plate to thereby make theuser visually recognize a virtual image. The head-mount type displaydevice includes two types, namely a transmissive type with which theuser can visually recognize the external sight in addition to thevirtual image, and a non-transmissive type with which the user is notallowed to visually recognize the external sight. The transmissivehead-mount type display device includes an optical transmissive type anda video transmissive type.

JP-A-5-268678 (Document 1) discloses a device control system in whichdisplay control data related to a control command having been receivedby a control target device for receiving the control command having beentransmitted from a controller is transmitted to the controller tothereby perform bidirectional data transmission/reception between thecontroller and the control target device. JP-A-2000-148381 (Document 2)discloses an input image processing method in which when a fingertip ofthe user as a predetermined mark is included in an imaged rangeequivalent to or larger than the visual field of the user, thecoordinate value of the fingertip is output to thereby perform a controlprocess corresponding to the position of the fingertip. JP-A-2013-205983(Document 3) discloses an information input method of operating acontrol target device based on an external sight imaged by a binocularcamera and the position and the posture of the hand of the user thusimaged. JP-A-2006-48628, JP-A-2005-69734, and JP-A-2000-163196 are otherexamples of the related art.

However, in the technology described in Document 1, in the case in whicha plurality of control target devices exists, it is unachievable for theuser to perform control with an intuitive operation in order for theuser to control either of the control target devices. Therefore, therehas been a problem that the usability needs to be improved. Further, inthe technology described in Document 2, the user needs to continue tofix the position of the fingertip until the control target device to theobject is selected, which incurs fatigue of the user. Therefore, therehas been a problem that the usability needs to further be improved.Further, in Document 3, although the control target device is operatedbased on the position and the posture of the hand, there has been aproblem that it is necessary to operate the control target device takingadditional information into consideration to thereby more intuitivelyoperate the control target device.

SUMMARY

An advantage of the invention is to solve at least a part of theproblems described above, and the invention can implemented as thefollowing aspects.

(1) An aspect of the invention provides a transmissive head-mount typedisplay device. The head-mount type display device includes an imagedisplay section adapted to display a virtual image, and capable oftransmitting an external sight, an object acquisition section adapted toobtain a selectable object located in a predetermined distance rangefrom the image display section, and a position of a specific objectincluded in the external sight, and a control section adapted to displayan object-correspondence virtual image associated with the objectobtained as the virtual image using the image display section, identifya change in the position of the specific object based on the position ofthe specific object obtained, select the object based on a relationshipbetween the change in the position of the specific object identified anda position of the object obtained, and display a specific check imageassociated with the object selected as the virtual image using the imagedisplay section. According to the head-mount type display device havingthis configuration, it is possible for the user to visually recognizethe object as an object of the operation and the control associated withthe change in the position of the specific object detected at the sametime without changing the eye direction, and to intuitively perform thecontrol of the object, and thus the convenience of the user is improved.

(2) In the head-mount type display device according to the aspectdescribed above, the object acquisition section may include an imagingsection adapted to image the external sight, and an image detectionsection adapted to detect and obtain the position of the specific objectand the object included in the external sight imaged, and the controlsection may determine control of the object selected, and then executesthe control of the object determined. According to the head-mount typedisplay device having this configuration, the specific object and theobject can be obtained by imaging, and at the same time, the user canvisually recognize the specific object and the object thus imaged, andtherefore, it is easy to recognize the object.

(3) In the head-mount type display device according to the aspectdescribed above, the control section may display theobject-correspondence virtual image associated in advance with acombination of the position of the specific object obtained and theobject obtained as the virtual image using the image display section.According to the head-mount type display device having thisconfiguration, it is possible for the user to recognize the change inthe position of the specific object necessary for executing the controlof the object as visual information, and thus, the usability of thehead-mount type display device for the user is improved.

(4) In the head-mount type display device according to the aspectdescribed above, the control section may display the virtual imagerepresenting the change in the position of the specific object necessaryfor executing the control of the object as the object-correspondencevirtual image associated in advance with the combination of the positionof the specific object obtained and the object obtained using the imagedisplay section. According to the head-mount type display device havingthis configuration, since it is possible for the user to recognize thecontent of the control of the object to subsequently be performed inaccordance with the change in the position of the own specific object asthe visual information, the usability for the user is improved.

(5) In the head-mount type display device according to the aspectdescribed above, the control section may display the virtual imagerepresenting a content of the control of the object to be executed asthe object-correspondence virtual image associated in advance with theobject obtained using the image display section. According to thehead-mount type display device having this configuration, since it ispossible for the user to recognize the content of the control of theobject to subsequently be performed in accordance with the change in theposition of the specific object as the visual information, the usabilityfor the user is improved.

(6) In the head-mount type display device according to the aspectdescribed above, in a case in which the position of the specific objecthaving been changed and the object overlap each other in the externalsight imaged, the control section may determine the control of theobject associated in advance with the object overlapping the position ofthe specific object having been changed. According to the head-mounttype display device having this configuration, since there is executedthe control of the object corresponding to the object overlapped by thespecific object after the position of the specific object has beenchanged, it is easy for the user to perform the modification of thespecific object corresponding to the control of the intended object.

(7) In the head-mount type display device according to the aspectdescribed above, the head-mount type display device may further includea distance identification section adapted to identify a distance betweenthe object obtained and the image display section, and the controlsection may set the object, which has been obtained, and has thedistance identified equal to or shorter than a threshold value, as theselectable object. According to the head-mount type display devicehaving this configuration, in the case in which a plurality of tentativeobjects has been detected in the taken image, those located near to theuser are set as the selectable objects. Therefore, since the user ismade to visually recognize these having a high possibility of beingselected by the user, the convenience of the user is enhanced.

(8) In the head-mount type display device according to the aspectdescribed above, that the head-mount type display device may furtherinclude a sound acquisition section adapted to obtain an external sound,and the control section may determine the control of the object based ona combination of the change in the position of the specific objectidentified and the sound obtained. According to the head-mount typedisplay device having this configuration, since the control of theobject is executed in accordance with the combination of the change inthe position of the specific object and the sound, it is possible forthe user to intuitively perform the control of a larger number ofobjects compared to the control of the object executed in accordanceonly with the change in the position of the specific object.

(9) In the head-mount type display device according to the aspectdescribed above, the object acquisition section may obtain positioninformation of at least one the objects from another device. Accordingto the head-mount type display device having this configuration, it ispossible to recognize the position of the selectable object in a rangewhich the user cannot visually recognize, and thus the usability for theuser is improved.

(10) In the head-mount type display device according to the aspectdescribed above, in a case in which the object obtained fails to beincluded in the external sight, the control section may display apositional relationship between the image display section and the objectobtained as the virtual image using the image display section. Accordingto the head-mount type display device having this configuration, it ispossible to visually recognize the positional relationship between theposition of the selectable object in the range, which the user cannotvisually recognize, and the user as a virtual image, and thus theusability for the user is further improved.

(11) In the head-mount type display device according to the aspectdescribed above, the control section may detect the acceleration of thespecific object, and identify the change in the position of the specificobject based on the acceleration of the specific object detected.According to the head-mount type display device having thisconfiguration, in the case in which the user executes the controlinstruction to the object selected, it is not required to operate aspecific place or to perform an action associated with the operation inthe range to be imaged, and thus, the usability for the user isimproved.

(12) In the head-mount type display device according to the aspectdescribed above, that the heat-mount type display device may furtherinclude a sound acquisition section adapted to obtain an external sound,and the control section may determine the control of the objectselected, based on the external sound obtained and a combination of thechange in the position of the specific object identified. According tothe head-mount type display device having this configuration, since thecontrol to the selected object is determined based on a plurality ofelements, namely the sound and the change in the position of thespecific object, the user can input a larger number of operations, andthus, the usability for the user is improved.

(13) In the head-mount type display device according to the aspectdescribed above, the control section may execute control of a controldevice associated with the object selected, and in a case in which theobject selected and the control device are obtained in the externalsight, the control section may display, as the virtual image using theimage display section, that the object selected and the control deviceare in a correspondence relationship. According to the head-mount typedisplay device having this configuration, it is possible to make thecorrespondence relationship between the selectable object detected andthe specific object detected be visually recognized, and thus, theusability for the user is improved.

(14) In the head-mount type display device according to the aspectdescribed above, the control section may display virtual images havingidentical shapes and identical colors using the image display section asthe virtual image representing that the object selected and the controldevice are in the correspondence relationship. According to thehead-mount type display device having this configuration, it is possibleto make the correspondence relationship between the selectable objectdetected and the specific object detected be visually recognized in amore articulate manner, and thus, the usability for the user isimproved.

(15) In the head-mount type display device according to the aspectdescribed above, the head-mount type display device may further includean identifying section adapted to identify an attribute of a user of theimage display section, and the control section may display at least oneof a virtual image associated with the object and the specific checkimage as the virtual image using the image display section so as tocorrespond to the attribute identified. According to the head-mount typedisplay device having this configuration, since the virtual imagedifferent by the user identified is generated in the image displaysection, the information meeting the needs of the user can be provided.

All of the constituents provided to each of the aspects of the inventiondescribed above are not necessarily essential, and in order to solve allor a part of the problems described above, or in order to achieve all ora part of the advantages described in the specification, it is possibleto arbitrarily make modification, elimination, replacement with anothernew constituent, partial deletion of restriction content on some of theconstituents. Further, in order to solve all or a part of the problemsdescribed above, or in order to achieve all or a part of the advantagesdescribed in the specification, it is also possible to combine some orall of the technical features included in one of the aspects of theinvention with some or all of the technical features included in anotherof the aspects of the invention to thereby form an independent aspect ofthe invention.

For example, an aspect of the invention can be implemented as a deviceprovided with some or all of the three elements, namely the imagedisplay section, the object acquisition section, and the controlsection. In other words, it is also possible for the device to includeor not to include the image display section. Further, it is alsopossible for the device to include or not to include the objectacquisition section. Further, it is also possible for the device toinclude or not to include the control section. It is also possible forthe image display section to, for example, display a virtual image, andto be capable of transmitting the external sight. It is possible for theobject acquisition section to obtain, for example, a selectable objectlocated in the predetermined distance range from the image displaysection, and the position of the specific object included in theexternal sight. It is possible for the control section to, for example,display an object-correspondence virtual image associated with theobject obtained as the virtual image using the image display section,identify a change in the position of the specific object based on theposition of the specific object obtained, select the object based on arelationship between the change in the position of the specific objectidentified and a position of the object obtained, and display a specificcheck image associated with the object selected as the virtual imageusing the image display section. Such a device can be realized as, forexample, a head-mount type display device, but can also be realized as adevice other than the head-mount type display device. According to suchan aspect of the invention, it is possible to solve at least one of avariety of problems such as improvement in operability andsimplification of the device, integration of the device, and enhancementof convenience of the user using the device. Some or all of thetechnical features of the head-mount type display device described aboveas each of the aspects of the invention can be applied to this device.

The invention can be implemented in various forms other than thehead-mount type display device. The invention can be implemented in theforms such as a display device, a method of controlling a display deviceor a head-mount type display device, a control system, a head-mount typedisplay system, a computer program for realizing the function of thecontrol system and the display device, a recording medium storing thecomputer program, and a data signal including the computer program andembodied in a carrier wave.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is an explanatory diagram showing an exterior configuration of anHMD.

FIG. 2 is a block diagram functionally showing a configuration of theHMD.

FIG. 3 is an explanatory diagram showing how image light is emitted byan image light generation section.

FIG. 4 is an explanatory diagram showing the flow of a device checkprocess.

FIG. 5 is an explanatory diagram showing an example of a visual field tobe visually recognized by the user in the case in which an operationobject has been detected.

FIG. 6 is an explanatory diagram showing an example of a visual field tobe visually recognized by the user in the case in which a determinationobject has been detected.

FIG. 7 is an explanatory diagram showing an example of a visual field tobe visually recognized by the user in the case in which a checkcompletion image has been displayed.

FIG. 8 is a block diagram functionally showing a configuration of acontrol system according to a second embodiment of the invention.

FIG. 9 is an explanatory diagram showing the flow of a control process.

FIG. 10 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case in which operationobjects have been detected.

FIG. 11 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case of a gesture detectionmode.

FIG. 12 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case in which an image ofthe gesture detection mode corresponding to a setting gesture has beendisplayed.

FIG. 13 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case in which apost-control image has been displayed.

FIG. 14 is a schematic diagram showing a configuration of a controlsystem according to a third embodiment of the invention.

FIG. 15 is a schematic diagram showing a part of information of acontrol device stored in a storage section of a server.

FIG. 16 is a block diagram functionally showing a configuration of anHMD according to a third embodiment.

FIG. 17 is an explanatory diagram showing the flow of a control processaccording to the third embodiment.

FIG. 18 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case in which a controldevice has been detected.

FIG. 19 is an explanatory diagram showing a visual field to be visuallyrecognized by the user in the case in which an image associated with acorresponding sequence has been displayed in the maximum image displayarea.

FIG. 20 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case in which an icon of animage has been selected.

FIG. 21 is an explanatory diagram showing an example of a visual fieldto be visually recognized by the user in the case in which positioninformation of a control apparatus has been obtained by iBeacon (aregistered trademark).

FIG. 22 is an explanatory diagram showing a visual field to be visuallyrecognized by the user in the case in which an image has been displayedin the maximum image display area in accordance with a controlinstruction performed.

FIGS. 23A and 23B are explanatory diagrams each showing an exteriorconfiguration of the HMD according to a modified example.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Then, some embodiments of the invention will be explained in thefollowing order based on some specific examples.

A. First Embodiment

A-1. Configuration of Head-Mount Type Display Device

A-2. Device Check Process

B. Second Embodiment

C. Third Embodiment

D. Modified Examples

A. First Embodiment

A-1. Configuration of Head-Mount Type Display Device

FIG. 1 is an explanatory diagram showing an exterior configuration of ahead-mount type display device 100 (HMD 100). The HMD 100 is a displaydevice to be mounted on the head, and is also called a head mounteddisplay (HMD). The HMD 100 according to the present embodiment is anoptical transmissive head-mount type display device allowing the user tovisually recognize a virtual image and at the same time visuallyrecognize an external sight directly. It should be noted that in thepresent specification, the virtual image to be visually recognized bythe user using the HMD 100 is also referred to as a “display image” forthe sake of convenience. Further, emission of the image light generatedbased on the image data is also referred to as “display of the image.”

The HMD 100 is provided with an image display section 20 for making theuser visually recognize the virtual image in the state of being mountedon the head of the user, and a control section 10 (a controller 10) forcontrolling the image display section 20.

The image display section 20 is a mounting body to be mounted on thehead of the user, and has a shape of a pair of glasses in the presentembodiment. The image display section 20 includes a right holdingsection 21, a right display drive section 22, a left holding section 23,a left display drive section 24, a right optical image display section26, a left optical image display section 28, a camera 61, an infraredLED 64, a TOF sensor 63, and a microphone 69. The right optical imagedisplay section 26 and the left optical image display section 28 aredisposed so as to be located in front of the right and left eyes of theuser, respectively, when the user wears the image display section 20.One end of the right optical image display section 26 and one end of theleft optical image display section 28 are connected to each other at aposition corresponding to the glabella of the user when the user wearsthe image display section 20.

The right holding section 21 is a member disposed so as to extend froman end portion ER, which is the other end of the right optical imagedisplay section 26, to a temporal region of the head of the user whenthe user wears the image display section 20. Similarly, the left holdingsection 23 is a member disposed so as to extend from an end portion EL,which is the other end of the left optical image display section 28, toa temporal region of the head of the user when the user wears the imagedisplay section 20. The right holding section 21 and the left holdingsection 23 hold the image display section 20 in the head of the user insuch a manner as the temples of the pair of glasses.

The right display drive section 22 and the left display drive section 24are disposed on the sides to be opposed to the head of the user when theuser wears the image display section 20. It should be noted thathereinafter the right holding section 21 and the left holding section 23are also collectively referred to simply as “holding sections,” theright display drive section 22 and the left display drive section 24 arecollectively referred to simply as “display drive sections,” and theright optical image display section 26 and the left optical imagedisplay section 28 are also collectively referred to simply as “opticalimage display sections.”

The display drive sections 22, 24 include liquid crystal displays 241,242 (hereinafter also referred to as “LCDs 241, 242”), projectionoptical systems 251, 252, and so on (see FIG. 2). The details of theconfiguration of the display drive sections 22, 24 will be describedlater. The optical image display sections 26, 28 as optical membersrespectively include light guide plates 261, 262 (see FIG. 2), anddimming plates. The light guide plates 261, 262 are each formed of alight transmissive resin material or the like, and guide the image lightoutput from the display drive sections 22, 24 to the eyes of the user,respectively. Each of the dimming plates is a thin-plate like opticalelement, and is disposed so as to cover the obverse side of the imagedisplay section 20, which is the side opposite to the side of the eyesof the user. The dimming plates protect the light guide plates 261, 262to suppress damages, adhesion of dirt, and so on to the light guideplates 261, 262, respectively. Further, by controlling the lighttransmittance of the dimming plate, an amount of the outside lightentering the eye of the user is controlled, and thus, the easiness ofthe visual recognition of the virtual image can be controlled. It shouldbe noted that the dimming plates can be eliminated.

The camera 61 is disposed at the position corresponding to the glabellaof the user when the user wears the image display section 20. Therefore,the camera 61 takes an image of the external sight as an outside view inthe eye direction of the user to obtain the taken image in the state inwhich the user wears the image display section 20 on the head. Thecamera 61 is a monocular camera, but can also be a stereo camera. Thecamera 61 corresponds to an imaging section in the appended claims.

The infrared LED 64 and the TOF sensor 63 is disposed at the positioncorresponding to the glabella of the user when the user wears the imagedisplay section 20, which is a similar position to the position of thecamera 61. The infrared LED 64 is a light emitting diode (LED) foremitting an infrared ray. The TOF sensor 63 detects the reflected light,which is the infrared light emitted by the infrared LED 64 and thenreflected by a specific object.

The microphone 69 is a device for obtaining an external sound. Themicrophone 69 is formed on the opposite side to the side opposed to theuser (the outer side) in the right display drive section 22 when theuser wears the image display section 20.

The image display section 20 further includes a connection section 40for connecting the image display section 20 to the control section 10.The connection section 40 includes a main body cord 48 to be connectedto the control section 10, a right cord 42, a left cord 44, and acoupling member 46. The main body cord 48 is branched into two cords toform the right cord 42 and the left cord 44. The right cord 42 isinserted into the housing of the right holding section 21 from a tipportion AP in the extending direction of the right holding section 21,and is connected to the right display drive section 22. Similarly, theleft cord 44 is inserted into the housing of the left holding section 23from a tip portion AP in the extending direction of the left holdingsection 23, and is connected to the left display drive section 24. Thecoupling member 46 is disposed at a branch point of the main body cord48, and the right cord 42 and the left cord 44, and has a jack to whichan earphone plug 30 is connected. A right earphone 32 and a leftearphone 34 extend from the earphone plug 30.

The image display section 20 and the control section 10 performtransmission of various signals via the connection section 40.Connectors (not shown) to be fitted with each other are provided to anend portion of the main body cord 48 on the opposite side to thecoupling member 46 and the control section 10, respectively. The controlsection 10 and the image display section 20 are connected to each otheror separated from each other in accordance with fitting/releasing of theconnector of the main body cord 48 and the connector of the controlsection 10. As the right cord 42, the left cord 44, and the main bodycord 48, there can be adopted, for example, metal cables or opticalfibers.

The control section 10 is a device for controlling the HMD 100. Thecontrol section 10 includes a determination key 11, a lighting section12, a display switching key 13, a track pad 14, a luminance switchingkey 15, direction keys 16, a menu key 17, and a power switch 18. Thedetermination key 11 detects a holding-down operation, and then outputsa signal for determining the content of the operation in the controlsection 10. The lighting section 12 gives notice of the operating stateof the HMD 100 with the lighting state of the lighting section 12. Asthe operating state of the HMD 100, there can be cited, for example, anON/OFF state of the power. As the lighting section 12, there is used,for example, an LED. The display switching key 13 detects a holding-downoperation, and then outputs, for example, a signal for switching thedisplay mode of the content moving image between a 3D mode and a 2Dmode. The track pad 14 detects the finger operation of the user on theoperation surface of the track pad 14, and then outputs a signalcorresponding to the detection content. As the track pad 14, there canbe adopted a variety of types of track pad such as an electrostatictrack pad, a pressure-detection track pad, or an optical track pad. Theluminance switching key 15 detects a holding-down operation to output asignal for increasing or decreasing the luminance of the image displaysection 20. The arrow keys 16 detect a holding-down operation to thekeys corresponding to up, down, right, and left directions to output asignal corresponding to the detection content. The power switch 18detects a sliding operation of the switch to thereby switch the poweringstate of the HMD 100.

FIG. 2 is a block diagram functionally showing a configuration of theHMD 100. As shown in FIG. 2, the control section 10 includes a storagesection 120, a power supply 130, a wireless communication section 132,an operation section 135, a CPU 140, an interface 180, a transmittingsection 51 (Tx 51), and a transmitting section 52 (Tx 52). The operationsection 135 receives the operation by the user, and includes thedetermination key 11, the display switching key 13, the track pad 14,the luminance switching key 15, the direction keys 16, the menu key 17,and the power switch 18.

The power supply 130 supplies each of the sections of the HMD 100 withthe electrical power. As the power supply 130, a secondary cell, forexample, can be used. The storage section 120 stores a variety ofcomputer programs. The storage section 120 includes a ROM, a RAM, and soon. Further, although the details will be described later, the storagesection 120 stores image data of the operation object detected in thecase of being included in the taken image taken by the camera 61, andimage data of the determination object to be the basis of thedetermination of performing the control. Further, the storage section120 also stores the display image to be displayed by the image displaysection 20 in the case in which the operation object or the like hasbeen detected.

The CPU 140 retrieves and then executes the computer programs stored inthe storage section 120 to thereby function as an operating system 150(OS 150), a display control section 190, a sound processing section 170,an image processing section 160, an image determination section 168, adistance measurement section 166, an image setting section 165, and aninterface 180.

The display control section 190 generates control signals forcontrolling the right display drive section 22 and the left displaydrive section 24. Specifically, the display control section 190 controlsto switch ON/OFF the drive of a right LCD 241 by a right LCD controlsection 211, to switch ON/OFF the drive of a right backlight 221 by aright backlight control section 201, to switch ON/OFF the drive of aleft LCD 242 by a left LCD control section 212, to switch ON/OFF thedrive of a left backlight 222 by a left backlight control section 202,and so on individually using the control signals. Thus, the displaycontrol section 190 controls generation and emission of the image lightby each of the right display drive section 22 and the left display drivesection 24. For example, the display control section 190 makes both ofthe right display drive section 22 and the left display drive section 24generate image light, makes either of them generate the image light, orinhibits the both from generating the image light.

The display control section 190 transmits the control signals to theright LCD control section 211 and the left LCD control section 212 viathe transmitting sections 51, 52, respectively. Further, the displaycontrol section 190 transmits the control signals to the right backlightcontrol section 201 and the left backlight control section 202,respectively.

The image processing section 160 obtains the image signal included inthe content. The image processing section 160 separates sync signalssuch as a vertical sync signal VSync and a horizontal sync signal HSyncfrom the image signal thus obtained. Further, the image processingsection 160 generates a clock signal PCLK using a phase locked loop(PLL) circuit or the like (not shown) in accordance with the periods ofthe vertical sync signal VSync and the horizontal sync signal HSync thusseparated. The image processing section 160 converts the analog imagesignal, from which the sync signals are separated, into a digital imagesignal using an A/D conversion circuit or the like (not shown).Subsequently, the image processing section 160 stores the digital imagesignal obtained by the conversion into a DRAM in the storage section 120frame by frame as the image data (RGB data) of the object image. Itshould be noted that it is also possible for the image processingsection 160 to perform image processing such as a resolution conversionprocess, various color correction processes of, for example, adjustmentof luminance and chromaticness, or a keystone distortion correctionprocess on the image data if necessary.

The image processing section 160 transmits each of the clock signalPCLK, the vertical sync signal VSync, the horizontal sync signal HSyncthus generated, and the image data stored in the DRAM in the storagesection 120, via each of the transmitting sections 51 and 52. It shouldbe noted that the image data transmitted via the transmitting section 51is also referred to as “right-eye image data,” and the image datatransmitted via the transmitting section 52 is also referred to as“left-eye image data.” The transmitting sections 51, 52 function astransmitters for serial transmission between the control section 10 andthe image display section 20.

The sound processing section 170 obtains a sound signal included in thecontent, amplifies the sound signal thus obtained, and then supplies theresult to a speaker (not shown) in the right earphone 32 and a speaker(not shown) in the left earphone 34 connected to the coupling member 46.It should be noted that in the case of, for example, adopting a Dolby(registered trademark) system, a process on the sound signal isperformed, and sounds different from each other and with, for example,modified frequencies are output respectively from the right earphone 32and the left earphone 34. The sound processing section 170 performs avariety of processes on the external sound obtained by the microphone69. Although the details will be described later, the sound processingsection 170 determines whether or not the external sound thus obtainedis a determination sound to be a basis of the determination ofperforming a device check process stored in the storage section 120 asthe variety of processes.

The image determination section 168 detects whether or not the imageidentical to the image data of the operation object or the determinationobject stored in the storage section 120 is included in the taken imageusing pattern matching or a statistical identification method. The imagedetermination section 168 detects the object, which is the same image asthe image data of the determination object thus detected, as thedetermination object. It should be noted that the image determinationsection 168 corresponds to an image detection section in the appendedclaims, and the determination object corresponds to a specific object inthe appended claims.

The distance measurement section 166 is a distance image sensor formeasuring the distance between the image display section 20 and thespecific object by calculating the time from when the infrared ray hasbeen emitted until the infrared ray is reflected by the specific objectand then received by the TOF sensor 63 by using the TOF (time of flight)method with respect to the reflected light of the infrared ray detectedby the TOF sensor 63. The distance measurement section 166 determineswhether or not the distance between the specific object representing thesame image as the image data of the operation object detected by theimage determination section 168 and the image display section 20 isequal to or shorter than a predetermined distance set in advance. Itshould be noted that the distance measurement section 166, the TOFsensor 63, and the infrared LED 64 correspond to a distanceidentification section in the appended claims.

The image setting section 165 sets a specific object, which has adistance from the image display section 20 equal to or shorter than apredetermined distance, and represents the same image as the image dataof the operation object stored in the storage section 120, as aselectable operation object. The image setting section 165 sets theselectable operation object, and then makes the image display section 20display an image associated with the operation object stored in thestorage section 120. In other words, a specific object, which has beendetected by the image determination section 168 as the same image as theimage data of the operation object, but has a distance from the imagedisplay section 20 exceeding the predetermined distance, is not set asthe selectable operation object. Further, the image setting section 165displays the image, which has previously been associated with thedetermination object thus detected, on the image display section 20. Theimage setting section 165 determines the control instruction, which hasbeen set based on the change in the position of the determination objectand the position of the operation object, and then performs the controlinstruction. It should be noted that the details of a variety of typesof images to be displayed on the image display section 20 will bedescribed later. The image setting section 165 corresponds to a controlsection in the appended claims. The selectable operation object in thepresent embodiment corresponds to a selectable object in the appendedclaims, and the control instruction set by the image setting section 165corresponds to control of the object in the appended claims.

The interface 180 is an interface for connecting various externalequipment OA to be a supply source of contents to the control section10. As the external equipment OA, there can be cited, for example, apersonal computer (PC), a cellular phone terminal, and a game terminal.As the interface 180, there can be used, for example, a USB interface, amicro USB interface, and an interface for a memory card.

The image display section 20 is provided with the right display drivesection 22, the left display drive section 24, the right light guideplate 261 as the right optical image display section 26, the left lightguide plate 262 as the left optical image display section 28, the camera61, the TOF sensor 63, the infrared LED 64, and the microphone 69.

The right display drive section 22 includes a receiving section 53 (Rx53), the right backlight control section 201 (right BL control section201) and the right backlight 221 (right BL 221) functioning as the lightsource, the right LCD control section 211 and the right LCD 241functioning as the display element, and a right projection opticalsystem 251. The right backlight control section 201 and the rightbacklight 221 function as the light source. The right LCD controlsection 211 and the right LCD 241 function as the display element. Itshould be noted that the right backlight control section 201, the rightLCD control section 211, the right backlight 221, and the right LCD 241are also collectively referred to as an “image light generationsection.”

The receiving section 53 functions as a receiver for serial transmissionbetween the control section 10 and the image display section 20. Theright backlight control section 201 drives the right backlight 221 basedon the control signal input to the right backlight control section 201.The right backlight 221 is a light emitter such as an LED orelectroluminescence (EL). The right LCD control section 211 drives theright LCD 241 based on the clock signal PCLK, the vertical sync signalVSync, the horizontal sync signal HSync, and the right-eye image datainput via the receiving section 53. The right LCD 241 is a transmissiveliquid crystal panel having a plurality of pixels arranged in a matrix.

The right projection optical system 251 is formed of a collimating lensfor converting the image light emitted from the right LCD 241 into alight beam in a parallel state. The right light guide plate 261 as theright optical image display section 26 guides the image light, which hasbeen output from the right projection optical system 251, to the righteye RE of the user while reflecting the image light along apredetermined light path. It should be noted that the right projectionoptical system 251 and the right light guide plate 261 are alsocollectively referred to as a “light guide section.”

The left display drive section 24 has substantially the sameconfiguration as that of the right display drive section 22. The leftdisplay drive section 24 includes a receiving section 54 (Rx 54), theleft backlight control section 202 (left BL control section 202) and theleft backlight 222 (left BL 222) functioning as the light source, theleft LCD control section 212 and the left LCD 242 functioning as thedisplay element, and a left projection optical system 252. The leftbacklight control section 202 and the left backlight 222 function as thelight source. The left LCD control section 212 and the left LCD 242function as the display element. It should be noted that the leftbacklight control section 202, the left LCD control section 212, theleft backlight 222, and the left LCD 242 are also collectively referredto as an “image light generation section.” Further, the left projectionoptical system 252 is formed of a collimating lens for converting theimage light emitted from the left LCD 242 into a light beam in aparallel state. The left light guide plate 262 as the left optical imagedisplay section 28 guides the image light, which has been output fromthe left projection optical system 252, to the left eye LE of the userwhile reflecting the image light along a predetermined light path. Itshould be noted that the left projection optical system 252 and the leftlight guide plate 262 are also collectively referred to as a “lightguide section.”

FIG. 3 is an explanatory diagram showing how the image light is emittedby the image light generation section. The right LCD 241 varies thetransmission of the light transmitted through the right LCD 241 bydriving the liquid crystal corresponding to each of the pixel positionsarranged in a matrix to thereby modulate the illumination light IL,which is emitted from the right backlight 221, into valid image light PLrepresenting the image. The same applies to the left side. It should benoted that although in the present embodiment, the backlight system isadopted as shown in FIG. 3, it is also possible to adopt a configurationof emitting the image light using a front light system or a reflectivesystem.

A-2. Device Check Process

FIG. 4 is an explanatory diagram showing the flow of the device checkprocess. In the device check process, when the control section 10detects a predetermined motion (hereinafter also referred to simply as a“gesture”) of the determination object (e.g., the finger of the user)after setting the selectable operation object (e.g., a meter of acontrol device), the control section 10 performs device check as thecontrol instruction base on the gesture.

Firstly, the camera 61 takes (step S10) an image of the external sightSC. It should be noted that the camera 61 continuously takes the imageof the external sight SC. In other words, the camera 61 takes the imageof the external sight SC as a moving image. The camera 61 transmits theimage data of the taken image thus taken to the image determinationsection 168 of the CPU 140. Then, the image determination section 168performs the image recognition process such as pattern matching on thetaken image taken by the camera 61 to thereby detect (step S12) whetheror not the image the same as the image data of the operation objectstored in the storage section 120 is included in the taken image. In thecase in which the image of the image data the same as the operationobject has not been detected in the taken image (NO in the step S12),the camera 61 repeats the process of the step S10 until the image of theimage data the same as the operation object is detected in the takenimage. In the case in which the image of the image data the same as theoperation object has been detected in the taken image in the process ofthe step S12 (YES in the step S12), the distance measurement section 166of the CPU 140 measures the distance between an object (hereinafter alsoreferred to simply as a “tentative object”) of the image representingthe operation object and the image display section 20 with the TOFmethod using the TOF sensor 63 and the infrared LED 64 at the same timewhen the image determination section 168 detects the tentative object inthe taken image. The distance measurement section 166 determines (stepS14) whether or not the distance measured between the tentative objectdetected by the image determination section 168 and the image displaysection 20 is equal to or shorter than a predetermined distance. In thecase in which it is determined that the distance measured between thetentative object and the image display section 20 exceeds thepredetermined distance (NO in the step S14), the image determinationsection 168 does not set the tentative object as the selectableoperation object, but the camera 61 repeats the process in the step S10and the subsequent steps. In the case in which it is determined in theprocess of the step S14 that the distance between the tentative objectand the image display section 20 is equal to or shorter than thepredetermined distance (YES in the step S14), the image setting section165 sets the tentative object detected by the image determinationsection 168 as the selectable operation object, and then makes the imagedisplay section 20 display (step S16) the image having been associatedwith the operation object and stored in the storage section 120.

FIG. 5 is an explanatory diagram showing an example of a visual field VRto be visually recognized by the user in the case in which operationobjects have been set. As shown in FIG. 5, as the operation objects thusset, the user visually recognizes three control devices MC, which aredevices different from the HMD 100, as the see-through external sightSC. Among the three control devices MC visually recognized by the user,the central one has a meter MT1 for indicating the voltage value ofanother device controlled by the control device MC, a meter MT2 forindicating the temperature of the another device, a button group BT as aplurality of buttons, and a lamp group LM including a plurality oflamps. Each of the buttons included in the button group BT is held downto be switched between an ON state and an OFF state to thereby controlthe another device. Each of the lamps of the lamp group LM is put ON inthe case in which the button included in the button group BT anddisposed on the upper side so as to correspond to the lamp is in the ONstate, and is put OFF in the case in which the button is in the OFFstate. It should be noted that the meter MT1 and the meter MT2 eachcorrespond to a selectable object in the appended claims.

In the present embodiment, since the storage section 120 stores theimage data of the meter MT1 and the meter MT2, the image determinationsection 168 detects the meter MT1 and the meter MT2 as the operationobjects. When the image determination section 168 sets the meter MT1 andthe meter MT2 as the operation objects, the image setting section 165makes an image IM1 and an image IM2 showing the respective positions ofthe meter MT1 and the meter MT2 be displayed on the maximum imagedisplay area PN of the image display section 20 as the images stored inthe storage section 120. It should be noted that the maximum imagedisplay area PN represents the maximum possible area for the image lightgeneration section to display an image, and is not visually recognizedby the user. Further, the image setting section 165 makes a text imageTX1 of “CHECKED?” be displayed in the maximum image display area PN asthe image, which is stored in the storage section 120, and is associatedwith at least one of the meter MT1 and the meter MT2 as the operationobjects thus set. The text image TX1 is an image for prompting the userto perform a check action of the numerical value indicated by the meterMT1 or the meter MT2.

When the meter MT1 and so on as the operation objects are set by theimage determination section 168, and the image such as the text imageTX1 is displayed by the image setting section 165 in the maximum imagedisplay area PN (step S16 shown in FIG. 4), the image determinationsection 168 detects (step S18) whether or not the image the same as theimage data of the determination object, which is used for determiningthe gesture and is stored in the storage section 120, is included in thetaken image. In the case in which the image of the determination objecthas not been detected in the taken image (NO in the step S18), the imagedetermination section 168 waits (step S18) for detection of thedetermination object. In the case in which the image of thedetermination object has been detected in the taken image (YES in thestep S18), the image setting section 165 of the CPU 140 makes thegesture instruction image, which is stored in the storage section 120and is used for instructing the gesture for the user to perform in orderto perform the check action prompted by the text image TX1, be displayed(step S20) in the maximum image display area PN.

FIG. 6 is an explanatory diagram showing an example of a visual field VRto be visually recognized by the user in the case in which adetermination object has been detected. FIG. 6 shows the state in whichthe first finger FF1 of the right hand HD1 of the user having beendetected as the determination object is held up. As shown in FIG. 6, inthe case in which the right hand HD1 as the determination object hasbeen detected, the image setting section 165 makes the gestureinstruction image, which shows the gesture to be performed forconfirming that the indications of the meters MT1, MT2 stored in thestorage section 120 are respectively lower than predetermined thresholdvalues, be displayed in the maximum image display area PN. It should benoted that the first finger FF1 of the right hand HD1 corresponds to aspecific object in the appended claims. The gesture instruction imagecorresponds to an object-correspondence virtual image in the appendedclaims.

The gesture instruction image is formed of an arrow image CS1 having aparabolic shape extending from the tip of the first finger FF1 of theright hand HD1 to the meter MT1, and a text image TX2 representing “OK!”which is a determination sound produced by the user. In the presentembodiment, when the determination sound of “OK!” has been detected bythe microphone 69 and the sound processing section 170 after the firstfinger FF1 has moved to a position overlapping the meter MT1 in thetaken image, the image setting section 165 performs the controlinstruction of the check action. It should be noted that in the devicecheck process stored in the storage section 120, it is arranged that inthe case in which both of the meters MT1, MT2 have been detected, thedevice check of the meter MT1 takes precedence. It is possible for theuser to visually recognize the external sight SC overlapping the textimages TX1, TX2 and the arrow image CS1 in a see-through manner. Thecontrol instruction of the check action corresponds to control of anobject determined in the appended claims.

When the image setting section 165 displays (step S20 shown in FIG. 4)the gesture instruction image in the maximum image display area PN, theimage determination section 168 waits (step S22) for detection of thegesture of the first finger FF1 along the arrow image CS1. The imagedetermination section 168 detects the gesture of the first finger FF1 byperforming the image recognition process such as pattern matching on theimage of the first finger FF1 in each of the frames of the taken imagescontinuously taken by the camera 61. In the case in which the gesture ofthe first finger FF1 along the arrow image CS1 has not been detected (NOin the step S22), the image determination section 168 continuously waits(step S22) for detection of the gesture of the first finger FF1 alongthe arrow image CS1. In the case in which the gesture of the firstfinger FF1 along the arrow image CS1 has been detected (YES in the stepS22), the image setting section 165 waits (step S24) for thedetermination sound of “OK!” represented by the text image TX2 to bedetected by the sound processing section 170 via the microphone 69 inthe state in which the position of the first finger FF1 after performingthe gesture overlaps the meter MT1 in the taken image. In the case inwhich no determination sound has been detected, or the case in which theposition of the first finger FF1 does not overlap the meter MT1 (NO inthe step S24), the image setting section 165 continuously waits (stepS24) for detection of the determination sound in the state in which theposition of the first finger FF1 and the meter MT1 overlap each other inthe taken image. In the case in which the determination sound of “OK!”has been detected in the state in which the position of the first fingerFF1 and the meter MT1 overlap each other in the taken image (YES in thestep S24), the image setting section 165 determines that the gesture forperforming the check action has been performed, makes a check completionimage to be displayed after completing the check action be displayed(step S26) in the maximum image display area PN, and then terminates thedevice check process. It should be noted that the check completion imagecorresponds to a specific check image in the appended claims.

FIG. 7 is an explanatory diagram showing an example of a visual field VRto be visually recognized by the user in the case in which the checkcompletion image has been displayed. FIG. 7 shows the visual field VRvisually recognized by the user when the check action by the user hasbeen detected, and then a text image TX3 of “MT1 CHECK OK!” as the checkcompletion image has been displayed. In the case in which the positionof the first finger FF1 and the meter MT1 overlap each other in thetaken image, the image setting section 165 makes an image IM3 of a solidcircle surrounding the meter MT1 be displayed in the maximum imagedisplay area PN in order to represent that the indication of the meterMT1 has been checked in addition to the image IM1 representing theposition of the meter MT1. It is possible for the user to visuallyrecognize the text image TX3 and the image IM3 displayed in the maximumimage display area PN to thereby confirm that the gesture has normallybeen processed, and the control instruction of the device check has beenperformed.

As explained hereinabove, in the HMD 100 according to the presentembodiment, the image determination section 168 detects the operationobject and the determination object included in the taken image, and theimage setting section 165 makes the images IM1, IM2 associated with theoperation object thus detected and the text image TX1 be displayed inthe maximum image display area PN, then determines the controlinstruction of the device check corresponding to the gesture of thedetermination object thus detected, and then performs the controlinstruction. Therefore, in the HMD 100 according to the presentembodiment, it is possible for the user to visually recognize theoperation object as an object of the operation and the controlassociated with the gesture to be performed by the user at the same timewithout changing the eye direction, and to intuitively perform thecontrol of the operation object, and thus the convenience of the user isimproved.

Further, in the HMD 100 according to the present embodiment, when theright hand HD1 as the determination object is detected, the imagesetting section 165 makes the arrow image CS1 representing the change inthe first finger FF1 be displayed in the maximum image display area PNas an image associated with the combination of the meters MT1, MT2 andthe first finger FF1 of the right hand HD1. Therefore, in the HMD 100according to the present embodiment, it is possible for the user torecognize the gesture necessary to perform the control instruction asvisual information, and thus, the usability of the HMD 100 for the useris improved.

Further, in the HMD 100 according to the present embodiment, in the casein which the sound processing section 170 has detected the determinationsound in the state in which the position of the first finger FF1 as thedetermination object and the meter MT1 as the operation object overlapeach other, the image setting section 165 makes the text image TX3 asthe check completion image be displayed in the maximum image displayarea PN. Therefore, in the HMD 100 according to the present embodiment,since there is performed the control instruction corresponding to theoperation object overlapped by the determination object after thegesture has been performed, it is easy for the user to perform thegesture corresponding to the control instruction intended. Further,since the control instruction is performed in accordance with thecombination of the gesture and the sound, it is possible for the user tointuitively perform a larger number of control instructions compared tothe control instructions performed in accordance with the gesture alone.

Further, in the HMD 100 according to the present embodiment, in the casein which the distance between the image display section 20 and thetentative object measured by the distance measurement section 166 isequal to or shorter than a predetermined distance, the image settingsection 165 sets the tentative object thus detected as the selectableoperation object. Therefore, in the HMD 100 according to the presentembodiment, in the case in which a plurality of tentative objects hasbeen detected in the taken image, those located near to the user are setas the selectable operation objects. Therefore, since the user is madeto visually recognize these having a high possibility of being selectedby the user, the convenience of the user is enhanced.

B. Second Embodiment

In the second embodiment, unlike the first embodiment, in the controlsystem 500 provided with an HMD 100 a and the control device 300,transmission and reception of the signal of the control instruction isperformed between the HMD 100 a and the control device 300. Thus, afterthe HMD 100 a has received the information for identifying the operationobject transmitted from the control device 300, the control of thecontrol device 300 is performed based on the control instruction thusinput in the HMD 100 a.

FIG. 8 is a block diagram functionally showing a configuration of acontrol system 500 according to the second embodiment. The controlsystem 500 is provided with the HMD 100 a and a control device 300. Itshould be noted that although FIG. 8 only shows one HMD 100 a and onecontrol device 300, the control system 500 can also be provided with aplurality of HMDs 100 a and a plurality of control devices 300, or it isalso possible to perform transmission and reception of information via adevice (e.g., a server) different from the HMD 100 a or the controldevice 300. The HMD 100 a in the second embodiment is provided with awireless communication section 132 included in the control section 10 inaddition to the configuration of the HMD 100 according to the firstembodiment. In FIG. 8, since the HMD 100 a is the same in configurationexcept the wireless communication section 132 as the HMD 100 accordingto the first embodiment (FIG. 2), the configuration of the HMD 100 a ispartially omitted from the graphical description.

The wireless communication section 132 of the HMD 100 a performswireless communication with other devices under a predetermined wirelesscommunication method such as wireless LAN or Bluetooth (registeredtrademark). The wireless communication section 132 transmits theinformation of the control instruction, which has been determined by thegesture of the user, to the control device 300. Further, the wirelesscommunication section 132 receives the information for identifying theoperation object, the determination object, the determination sound, thegesture, and so on from the control device 300. It should be noted thatthe wireless communication section 132 in the second embodimentcorresponds to a second communication section in the appended claims.

The control device 300 is provided with a wireless communication section330, a storage section 320, and a CPU 310. The wireless communicationsection 330 performs wireless communication with the HMD 100 a under thepredetermined wireless communication method such as the wireless LAN orthe Bluetooth. It should be noted that the wireless communicationsection 330 in the second embodiment corresponds to a firstcommunication section in the appended claims. The storage section 320 isconstituted by, for example, a ROM, a RAM, a DRAM, or a hard disk. Thestorage section 320 stores the control content associated with thecontrol instruction for controlling the control device 300. Further, thestorage section 320 stores the image data of the operation objectdetected in the case of being included in the taken image taken by thecamera 61, and the image data of the determination object to be thebasis of the determination of performing the control. It should be notedthat the storage section 320 in the second embodiment corresponds to anobject identification section in the appended claims.

The CPU 310 retrieves and then executes a computer program stored in thestorage section 320 to thereby function as an information processingsection 312 and a control processing section 314. The informationprocessing section 312 processes the information of identifying thecontrol instruction and the operation object transmitted and receivedbetween the information processing section 312 and the wirelesscommunication section 132 of the HMD 100 a via the wirelesscommunication section 330. The control processing section 314 controlsthe control device 300 so as to correspond to the control instruction,which has been processed by the information processing section 312,based on the information of the control instruction. It should be notedthat the control device 300 in the second embodiment corresponds to acontrol execution section in the appended claims.

FIG. 9 is an explanatory diagram showing the flow of a control process.The process corresponding to the steps from S30 to S36 in the controlprocess is the same as the process corresponding to the steps from S10to S18 except the process of identifying the distance between the imagedisplay section 20 and the operation object in the device check process(FIG. 4) according to the first embodiment. Therefore, in thedescription of the second embodiment, the process corresponding to thesteps from S30 to S36 in the control process will briefly be explained.

In the control process, when the external sight SC is imaged (step S30)by the camera 61, and the operation object has been detected in thetaken image (YES in the step S32), the image setting section 165 makesthe image associated with the operation object be displayed in themaximum image display area PN.

FIG. 10 is an explanatory diagram showing an example of a visual fieldVR to be visually recognized by the user in the case in which operationobjects have been detected. FIG. 10 shows the external sight SC to bevisually recognized by the user in a living room, and images IM4, IM5displayed in the maximum image display area PN. As shown in FIG. 10, theuser visually recognizes a remote controller RC1 of a television set anda switch SW as a switch of a room illumination, which are the operationobjects set by the image determination section 168, and the image IM4associated with the remote controller RC1 and the image IM5 associatedwith the switch SW both stored in the storage section 320. It should benoted that the remote controller RC1 and the switch SW each correspondto an object in the appended claims.

In the state in which the images IM4, IM5 are displayed (step S34 shownin FIG. 9), in the case in which the determination object has beendetected by the image determination section 168 (YES in the step S36),the image setting section 165 makes the image of a gesture detectionmode representing the state, in which the gesture of the user can bedetected, be displayed (step S38) in the maximum image display area PN.

FIG. 11 is an explanatory diagram showing an example of a visual fieldVR to be visually recognized by the user in the case of the gesturedetection mode. As shown in FIG. 11, the image determination section 168detects the right hand HD1 as the determination object, and the imagesetting section 165 displays an IM6 associated with the remotecontroller RC1 and then displays an arrow image CS2 and an image IM7both associated with the switch SW. The image IM6 is an image includinga plurality of icons representing contents to be controlled if theposition of the first finger FF1 of the right hand HD1 overlaps theicons. For example, when the “ON/OFF” icon among the icons displayed isselected in accordance with a change in the position of the finger FF1,the control processing section 314 performs the control of switchingbetween the ON state and the OFF state of the power of the televisionset TV1. Further, the arrow image CS2 shown in FIG. 11 is an imagerepresenting the gesture, which should be performed by the user in orderto perform the control of selecting the switch SW. An image IM7 is animage representing the control content to be performed when the gestureof the first finger FF1 along the arrow image CS2 has been performed.Specifically, the image IM7 represents that the switch SW is selectedwhen the first finger FF1 of the user moves along the arrow image CS2 soas to surround the switch SW.

When the image setting section 165 displays (step S38 shown in FIG. 9)the image of the gesture detection mode in the maximum image displayarea PN, the image determination section 168 waits (step S40) fordetection of the gesture of the right hand HD1 for performing thecontrol of the remote controller RC1 or the switch SW. In the case inwhich the gesture of the right hand HD1 has not been detected (NO in thestep S40), the image determination section 168 continuously waits (stepS40) for detection of the gesture of the right hand HD1. In the case inwhich the gesture of the right hand HD1 has been detected (YES in thestep S40), the image setting section 165 performs (step S42) the controlcorresponding to the detected gesture, which is a gesture having beendetected. The image setting section 165 determines (step S44) whether ornot there exists a setting gesture, which is a gesture corresponding toexecution of another control set subsequently to the control having beenperformed corresponding to the detected gesture. In the case in which ithas been determined that the setting gesture exists (YES in the stepS44), the image setting section 165 makes the image of the gesturedetection mode corresponding to the setting gesture be displayed (stepS38) in the maximum image display area PN.

FIG. 12 is an explanatory diagram showing an example of a visual fieldVR to be visually recognized by the user in the case in which the imageof the gesture detection mode corresponding to the setting gesture hasbeen displayed. FIG. 12 shows an image IM8, which is a gestureinstruction image corresponding to the setting gesture to be performedafter the motion of the first finger FF1 along the arrow image CS2 (FIG.11) has been detected as the detected gesture. The image IM8 is an imagerepresenting the control of setting the switch SW to the ON state in thecase in which the first finger FF1 is moved in the upward direction, andsetting the switch SW to the OFF state in the case in which the firstfinger FF1 is moved in the downward direction.

When the image setting section 165 displays (step S38 shown in FIG. 9)the image of the gesture detection mode in the maximum image displayarea PN, and the gesture of setting the switch SW to the ON state or theOFF state has been detected (YES in the step S40), the image settingsection 165 performs (step S42) the control corresponding to thedetected gesture. In the case in which there exists the setting gesturecorresponding to the control to be performed subsequently to the controlhaving been performed (YES in the step S44), the process correspondingto the step S38 and the subsequent steps. In the case in which thesetting gesture corresponding to the control to be performedsubsequently to the control having been performed does not exist in theprocess of the step S44 (NO in the step S44), the image setting section165 makes a post-control image, which represents that the controlcorresponding to the detected gesture has been performed, be displayed(step S46) in the maximum image display area PN.

FIG. 13 is an explanatory diagram showing an example of a visual fieldVR to be visually recognized by the user in the case in which thepost-control image has been displayed. FIG. 13 shows the state in whicha text image TX4 as the post-control image is displayed after thesetting gesture of setting the switch SW as the detected gesture to theOFF state has been detected. As shown in FIG. 13, when the switch SW isset to the OFF state by the control processing section 314, the textimage TX4 of “SWITCH WAS TURNED OFF” for informing the user of the factthat the switch SW is set to the OFF state by the control processingsection 314 is displayed by the image setting section 165 in the maximumimage display area PN.

When the text image TX4 of the post-control image is displayed (step S46shown in FIG. 9) in the maximum image display area PN, the image settingsection 165 displays the text image TX4 in the maximum image displayarea PN for a predetermined period of time. While the text image TX4 isdisplayed in the maximum image display area PN, the operation section135 waits (step S48) for detection of a predetermined operation forterminating the control process using the operation object and so onhaving been detected in the taken image. In the case in which theoperation section 135 has failed to detect the predetermined operationfor terminating the control process while the text image TX4 isdisplayed in the maximum image display area PN (NO in the step S48), theimage setting section 165 sets the text image TX4 to a nondisplay state,and then the process corresponding to the step S32 and the subsequentsteps. In the case in which the operation section 135 detects thepredetermined process of terminating the control process in the processof the step S48 (YES in the step S48), the control section 10 terminatesthe control process. It should be noted that in the second embodiment,by the operation section 135 detecting the predetermined operation, thecontrol section 10 can terminate the control process even in themid-flow of the control process.

As explained hereinabove, in the HMD 100 a according to the secondembodiment, the image setting section 165 makes the image IM8representing the control of the switch SW to the ON state and the OFFstate, which is performed in accordance with the direction of the motionof the first finger FF1 of the right hand HD1 along the verticaldirection, be displayed in the maximum image display area PN as shown inFIG. 12. Therefore, in the HMD 100 a according to the second embodiment,since it is possible for the user to recognize the content of thecontrol to be subsequently performed in accordance with the gesture ofthe user as visual information, the usability for the user is improved.

C. Third Embodiment

The third embodiment is mainly different from the first and secondembodiments in that the position information of the operation object isobtained as optical information or via wireless communication, and thatthe change in the position of the determination object is identified bydetecting the acceleration of the determination object.

FIG. 14 is a schematic diagram showing a configuration of a controlsystem 500 b according to the third embodiment. The control system 500 bis provided with an HMD 100 b, a server 400, and a control device 300 b.The HMD 100 b is provided with an image display section 20 b, a controlsection 10 b, and an external sensor 70 as the determination object. Theexternal sensor 70 incorporates an acceleration sensor for detecting themotion of the external sensor 70. The external sensor 70 transmits theacceleration thus detected to the control section 10 b using acommunication device such as Bluetooth. The control section 10 bcalculates the change in the position of the external sensor 70 based onthe acceleration of the external sensor 70 thus transmitted. The controlsection 10 b controls the control device 300 b based on the change inthe position of the external sensor 70 thus calculated. The imagedisplay section 20 b does not include the TOF sensor 63 and the infraredLED 64 in the first and second embodiments. It should be noted thatalthough in the present embodiment, as shown in FIG. 1, the explanationis presented assuming the control device 300 b as a television set, anillumination device or the like other than the television set can alsobe adopted as the control device 300 b in other embodiments.

The server 400 includes storage section 420 for storing the informationof the control device 300 b such as a control apparatus on which acontrol instruction is performed by the gesture detected, and theoperation object for providing the control instruction to the controlapparatus. It should be noted that the gesture in the third embodimentincludes the change in the position of the external sensor 70. Theserver 400 transmits at least a part of the information stored in thestorage section 420 to the HMD 100 b so as to correspond to a signaltransmitted from the HMD 100 b. The wireless communication section 132 bof the control section 10 b of the HMD 100 b receives the informationhaving been transmitted from the server 400.

FIG. 15 is a schematic diagram showing a part of the information of thecontrol device stored in the storage section 420 of the server 400. Thestorage section 420 stores the information of a plurality of controldevices 300 b. For example, as shown in FIG. 15, the television set asone of the control devices 300 b is associated with a three-dimensionalmodel and a position coordinate with respect to each of a controlapparatus for displaying programs to be put on television and acontroller for operating the control apparatus. Further, the televisionset of the control device 300 b is associated with the displayinformation. The three-dimensional model shown in FIG. 15 isthree-dimensional image data stored with respect to each of the controlapparatus and the controller. The position coordinate represents theplace at which each of the control apparatus and the controller islocated in the case of taking the HMD 100 b as a reference when the HMD100 b receives the information from the server 400. The display image isan example of the content of the image to be displayed on the imagedisplay section 20 b in the case in which the controller has beendetected.

In the third embodiment, the storage section 320 b of the control device300 b stores each of procedures in sequence control to be performedusing a plurality of procedures. For example, in the case in which thecontrol device 300 b is the television set, and a program to be put ontelevision with later time designated is recorded (also described simplyas “timer recording is performed”), if a plurality of procedures such astime and date, and the channel of the program to be put on television iscorrectly input, the recording reservation is completed.

FIG. 16 is a block diagram functionally showing a configuration of theHMD 100 b according to the third embodiment. A storage section 120 b ofthe third embodiment stores personal information for identifying theuser of the HMD 100 b unlike the storage section 120 of the first andsecond embodiments. As shown in FIG. 16, a CPU 140 b of the controlsection 10 b of the third embodiment does not include the distancemeasurement section 166 in the first and second embodiments, andincludes a position identification section 163 and an identifyingsection 161, When powering on the HMD 100 b, the identifying section 161makes an image for identifying the user of the HMD 100 b be displayed inthe maximum image display area PN. Subsequently, the identifying section161 identifies the user of the HMD 100 b based on the operation receivedby the operation section 135. As the identification method of the user,for example, the following steps are performed in some cases. Firstly,the identifying section 161 makes an image, which is used for making aprevious user of the HMD 100 b registered in advance be selected, bedisplayed in the maximum image display area PN. Subsequently, theidentifying section 161 makes an image for requiring input of a passwordhaving been stored in the storage section 320 b be displayed in themaximum image display area PN so as to correspond to the user selectedby the operation received by the operation section 135, and thenidentifies the user of the HMD 100 b if the password thus required isinput.

The position identification section 163 identifies the positionalrelationship between the image display section 20 b, the controlapparatus, and the operation object based on the position information ofthe control apparatus and the position information of the controller asthe operation object obtained from the storage section 420 of the server400 via the wireless communication section 132 b. It should be notedthat the position of the image display section 20 b is identified basedon the intensity of a radio wave, a time difference between radio wavestransmitted and received be a plurality of base stations or the like,and so on. Further, the position identification section 163 identifiesthe change in the position of the external sensor 70 based on theacceleration detected by the external sensor 70. It should be noted thatthe external sensor 70 corresponds to a specific object in the appendedclaims.

FIG. 17 is an explanatory diagram showing the flow of a control processaccording to the third embodiment. The control process according to thethird embodiment is a process in which the control instruction of thecontrol apparatus identified in position based on the change in theexternal sensor 70 detected.

In the control process according to the third embodiment, firstly, theidentifying section 161 identifies (step S50) the user of the HMD 100 b.In the third embodiment, the type of the control device 300 b to bedetected is different by the user thus identified. Specifically,although in one user, a specific control device is detected as an objecton which the control instruction can be performed, in another user, thespecific control device is determined as an object on which the controlinstruction cannot be performed, and the specific control device is notdetected.

When the user is identified, the position identification section 163identifies (step S51) the position of the control device located withina predetermined range from the user wearing the image display section 20b from the server 400 via the wireless communication section 132 b.Then, the camera 61 takes (step S52) an image of the external sight.Then, the position identification section 163 determines (step S54)whether or not the control device 300 b, the position of which has beenidentified within the predetermined range from the user, has beendetected. In the case in which the position identification section 163fails to detect the control device 300 b (NO in the step S54), theposition identification section 163 continuously waits (step S51) forthe identification of the position of the control device 300 b locatedwithin the predetermined range from the user. It should be noted that itis sufficient for the control device 300 b, which is to be an object ofdetection on presence or absence in the third embodiment, to be at leasteither one of the control apparatus and the controller.

In the case in which the control device 300 b has been detected in theprocess of the step S54 (YES in the step S54), the image setting section165 makes the image representing the position of the control device 300b be displayed (step S56) in the maximum image display area PN. Itshould be noted that in the third embodiment, the positionidentification section 163 uses the pattern matching or the like withrespect to the imaging range of the place identified as the position ofthe control device 300 b with reference to the three-dimensional modelof the control device 300 b stored in the storage section 420 of theserver 400 to thereby improve the accuracy of the position of thecontrol device 300 b with respect to the user. It should be noted thatin another embodiment, the position identification section 163 does notneed to identify the position of the control device 300 b using imagerecognition.

FIG. 18 is an explanatory diagram showing an example of a visual fieldVR to be visually recognized by the user in the case in which thecontrol device 300 b has been detected. As shown in FIG. 18,corresponding markers MK1, MK2, and MK3 representing three combinationsas the control devices 300 b thus detected, and a birds-eye view VI aredisplayed in the maximum image display area PN. In the example shown inFIG. 18, the three control devices 300 b thus detected are as follows:

(1) the television set TV1 as the control apparatus, and the remotecontroller RC1 as the controller;

(2) an illumination LT as the control apparatus, and the switch SW asthe controller; and

(3) an air conditioner AC as the control apparatus, and a remotecontroller RC2 as the controller.

The control apparatus and the controller in each of the three controldevices are shown so as to be associated with each other using thecorresponding markers MK1, MK2, or MK3 as images having the same shapeand the same color. For example, the corresponding markers MK1 aredisplayed in the maximum image display area PN in the vicinity of thetelevision set TV1 as the control apparatus MC and the vicinity of theremote controller RC1 as the controller 10. It should be noted that thecorresponding markers having the same shape in the third embodiment arereferred to as the corresponding markers having the same shape even ifthe corresponding markers displayed are different in size from eachother.

The birds-eye view VI is a miniature showing an outline of thepositional relationship between the user, the imaging range ey of thecamera 61, and the remote controller RC2, which is detected and is notincluded in the imaging range, viewed from the above of the usercentered on the user wearing the image display section 20 b. In the casein which a part of the control device 300 b, the position of which hasbeen identified by the position identification section 163, is notincluded in the imaging range of the camera 61, the image settingsection 165 makes the birds-eye view VI, which shows the positionalrelationship between the user and the part of the control device 300 bnot included in the imaging range, be displayed in the maximum imagedisplay area PN. As shown in the birds-eye view VI, the remotecontroller RC2 of the air conditioner AC exists on the right side of theuser.

When the image representing the position of the control device 300 b isdisplayed (step S56 shown in FIG. 17) in the maximum image display areaPN, the position identification section 163 determines (step S58) thepresence or absence of the detection of the change in the position ofthe external sensor 70 as the determination object. In the case in whichthe position identification section 163 fails to detect the change inthe position of the external sensor 70 (NO in the step S58), theposition identification section 163 continuously waits (step S58) forthe detection of the change in the position of the external sensor 70 asthe determination object.

In the case in which the change in the position of the external sensor70 as the determination object has been detected in the process of thestep S58 (YES in the step S58), the image setting section 165 executes(step S60) the control instruction of the control device 300 bcorresponding to the change in the position of the external sensor 70thus detected. As the control instruction corresponding to the change inthe position of the external sensor 70, the control instructionsdifferent from each other are set to the respective control devices 300b in some cases. For example, in the case of selecting the remotecontroller RC1 of the television set TV1, when the change in theposition where the circular shape of the corresponding marker associatedtherewith has been detected from the external sensor 70, the imagesetting section 165 makes a menu image of the television set TV1corresponding to the image IM4 be displayed in the maximum image displayarea PN. It should be noted that in another embodiment, in the case inwhich the external sensor 70 is included in the taken image taken by thecamera 61, it is possible for the image setting section 165 to executethe control instruction to the control device 300 b based on theacceleration of the external sensor 70 thus detected and the change inthe external sensor 70 thus imaged.

When the control instruction corresponding to the change in the positionof the external sensor 70 as the operation object thus detected isexecuted (step S60 shown in FIG. 17), the image setting section 165makes the image, on which the control has been performed, be displayed(step S62) in the maximum image display area PN. In the thirdembodiment, since the procedures of the sequence control performed witha plurality of procedures are stored in the storage section 320 b of thecontrol device 300 b, after the image on which the control has beenperformed is displayed in the maximum image display area PN, the imagesetting section 165 determines (step S64) whether or not thecorresponding sequence, which has been set as the subsequent procedureto the sequence control, exists in the storage section 320 b of thecontrol device 300 b via the wireless communication section 132 b. Inthe case in which the image setting section 165 determines that thecorresponding sequence exists (YES in the step S64), the image settingsection 165 makes the image associated with the corresponding sequencebe displayed (step S66) in the maximum image display area PN.

FIG. 19 is an explanatory diagram showing a visual field VR to bevisually recognized by the user in the case in which an image associatedwith the corresponding sequence has been displayed in the maximum imagedisplay area PN. FIG. 19 shows the state in which the menu of the imageIM4 associated with the remote controller RC1 shown in FIG. 18 isselected, and then an image IM9 representing the screen for setting thechannel in performing the timer recording is displayed in the maximumimage display area PN. When the remote controller RC1 is operated toperform the timer recording, there is performed the sequence controlalong the following order, setting of the channel to be recorded, timeand date when the timer recording is performed, and whether or notperiodic weekly timer recording is performed.

If the image IM9 is displayed in the maximum image display area PN, andthe external sensor 70 detects an action of drawing a number as theprocess of the step S58 shown in FIG. 17 (YES in the step S58), theimage setting section 165 sets (step S60) the number corresponding tothe action thus detected as the channel, and makes the number set in the“**” part of the image IM9 be displayed (step S62). For example, whenthe user moves the external sensor 70 so as to draw a number of “B” inan empty space, the “B” channel is set as the channel to be recorded.Further, when the external sensor 70 detects the change of being swungleftward along the arrow shown in the image IM9, the image settingsection 165 makes the previous image to the setting of the channel to berecorded in the sequence control be displayed in the maximum imagedisplay area PN. Further, in the state in which the number of thechannel to be recorded is input, when the external sensor 70 detects thechange of being swung rightward along the arrow shown in the image IM9,the image setting section 165 sets the channel to be recorded and makesthe next image to the setting of the channel to be recorded in thesequence control be displayed in the maximum image display area PN.Subsequently, the process in the step S64 and the subsequent steps isperformed in a similar manner.

As explained hereinabove, in the HMD 100 b according to the thirdembodiment, the wireless communication section 132 b of the controlsection 10 b receives the position information of the control device 300b stored in the storage section 420 of the server 400. Further,therefore, in the HMD 100 b according to the third embodiment, theposition of the control device 300 b in a range, which the user cannotvisually recognize, can be recognized, and thus, the usability for theuser is improved.

Further, in the HMD 100 b according to the third embodiment, the imagesetting section 165 makes the birds-eye view VI, which represents thepositional relationship between the user wearing the image displaysection 20 b and the remote controller RC2 detected, be displayed in themaximum image display area PN as shown in FIG. 19. Therefore, in the HMD100 b according to the third embodiment, the positional relationshipbetween the position of the control device 300 b in a range, which theuser cannot visually recognize, and the user can visually be recognizedas an image, and thus, the usability for the user is further enhanced.

Further, in the HMD 100 b according to the third embodiment, theexternal sensor 70 detects the acceleration as the change in theposition of the external sensor 70. The image setting section 165executes the control instruction to the control device 300 b based onthe change in the position of the external sensor 70 thus detected.Therefore, in the HMD 100 b according to the third embodiment, in thecase in which the user executes the control instruction to the controldevice 300 b, it is not required to operate a specific place or toperform an action associated with the operation in the range to beimaged, and thus, the usability for the user is improved.

Further, in the HMD 100 b according to the third embodiment, as shown inFIG. 18, the image setting section 165 makes the position of the controlapparatus detected and the position of the controller of the controlapparatus detected be displayed in the maximum image display area PN soas to be associated with each other using the corresponding markers asthe images having the same shape and the same color. Therefore, in theHMD 100 b according to the third embodiment, the correspondencerelationship between the control apparatus detected and the controllerof the control apparatus detected can be made to be visually recognized,and thus, the usability for the user is improved.

Further, in the HMD 100 b according to the third embodiment, theidentifying section 161 identifies the user of the HMD 100 b, and theimage setting section 165 makes the image related to the control device,on which the control instruction of the user thus identified can beexecuted, be displayed in the maximum image display area PN. Therefore,in the HMD 100 b according to the third embodiment, since the imagedifferent by the user identified is displayed in the maximum imagedisplay area PN, the information meeting the need of the user can beprovided.

D. Modified Examples

It should be noted that the invention is not limited to the embodimentsdescribed above, but can be implemented in various forms within thescope or the spirit of the invention, and the following modifications,for example, are also possible.

D1. Modified Example 1

Although in the embodiments described above, the image setting section165 displays the images corresponding to the operation object thus setand the determination object thus detected, and the gesture instructionimages in the maximum image display area PN, it is not necessarilyrequired to display these images. For example, it is also possible forthe image setting section 165 to display the image corresponding to theoperation object thus set, and then perform the control corresponding tothe gesture detected without displaying the image corresponding to thedetermination object and the gesture instruction image.

Although in the present embodiment, the distance measurement section 166measures the distance between the image display section 20 and thetentative object using the infrared LED 64 and the TOF sensor 63, it isnot necessarily required to measure the distance, and further, thedistance can also be measured using other methods. For example, it isalso possible for the image setting section 165 to set all of thetentative objects detected by the image determination section 168 as theselectable operation objects irrespective of the distance between theimage display section 20 and the operation object. Further, it is alsopossible for the distance measurement section 166 to compare the imagesof the operation objects imaged by a stereo camera with each other tothereby measure the distance between the image display section 20 andthe operation object.

Although in the embodiments described above, the image determinationsection 168 detects the first finger FF1 of the right hand HD1 as thedetermination object, the determination object to be detected canvariously be modified. For example, the determination object can be thecontrol section 10, or can be a ballpoint pen or a stylus pen foroperating the track pad 14 of the control section 10. Further, thedetermination object can also be a rod-like instrument for pointing onthe extension of the first finger FF1, a specific shape of light in alaser pointer, an instrument (e.g., a flashlight) with a specific shapehaving a light emitting section, or the like. The presence or absence ofthe detection of the determination object can be determined by thepresence or absence of the light emission so that the penlight in thelight emission state is detected as the determination object while thepenlight in the non-light emission state is not detected as thedetermination object. In the case in which whether or not the penlightis detected as the determination object is determined based on thepresence or absence of the light emission, the detection of the penlightas the determination object higher in accuracy is performed. Byoperating the operation section 135, the determination object canarbitrarily be set by the user. Further, the determination object can bea wristwatch-type wearable device to be mounted on an arm of the user, aring-like wearable device to be mounted on a finger of the user, or thelike.

Further, the number of the determination objects to be detected is notlimited to one, but can also be equal to or larger than two. Forexample, it is also possible that two fingers are detected as thedetermination objects, and the distance between the two fingers or theshapes of the two fingers are recognized instead of the gesture. Forexample, the gesture corresponding to pinch-in or pinch-out forperforming the control of magnification or reduction of the displayimage can be detected due to the motion of the two fingers detected asthe determination objects. As a similar operation by the two fingers, bysetting the knob of the volume as the determination object and takingthe knob between the two fingers to turn the knob, for example,clockwise, the volume to be output can be varied. Further, for example,in the case of operating the keyboard, as an alternate means of such anoperation as to input a capital alphabet when selecting a specificcharacter while holding down the Shift key, it is possible to detect avariety of types of input by detecting a relative motion of one fingerwith respect to the other finger in the positional relationship betweenthe two fingers detected.

Although in the first embodiment described above, the image settingsection 165 determines the check action of the meter MT1 using thecombination of the gesture and the determination sound detected afterperforming the gesture, the determination sound is not necessarilyrequired to be detected, and it is possible to determine the checkaction of the meter MT1 using only the gesture. Further, although theimage setting section 165 detects the determination sound afterperforming the gesture to thereby determine the check action of themeter MT1, the timing of detecting the determination sound is notlimited to the period after performing the gesture, but can be in theperiod before the gesture, and it is also possible to determine thecheck action using the determination sound detected while the gesture isbeing detected.

The object-correspondence virtual image in the appended claims broadlymeans the image displayed before some control or processing such as aselective image displayed in the maximum image display area PN, or animage suggesting the gesture necessary for proceeding to the nextprocess. Further, the specific check image in the appended claimsbroadly means the image provided after some process or control based onthe object-correspondence virtual image has been performed. The specificcheck image corresponds to the image for announcing the process and thecontrol having been performed, or includes an image representing thatthe reception of some process has been confirmed. Therefore, in somecases, the same image corresponds to the object-correspondence virtualimage, and at the same time corresponds to the specific check image. Incontrast, even in the case of the same image, there are some cases inwhich the image corresponds to the object-correspondence virtual image,and some cases in which the image does not correspond to theobject-correspondence virtual image in accordance with the circumstancesin which the image is displayed.

D2. Modified Example 2

Although in the description of the second embodiment, there is explainedthe control in which the switch SW is set to the OFF state, in thismodified example, the control performed in the case in which the icondisplayed in the image IM6 corresponding to the remote controller RC1 isselected will be explained. FIG. 20 is an explanatory diagram showing anexample of the visual field VR to be visually recognized by the user inthe case in which the icon of the image IM6 has been selected. As shownin FIG. 20, the position of the first finger FF1 of the right hand HD1overlaps the icon “ON/OFF” in the image IM6. In this case, the controlprocessing section 314 switches between the ON state and the OFF stateof the power of the television set TV1. In the example shown in FIG. 20,since the television set TV1 has been powered OFF before the position ofthe first finger FF1 overlaps the icon, when the position of the firstfinger FF1 overlaps the icon “ON/OFF” in the taken image, the controlprocessing section 314 powers ON the television set TV1. Although thegesture image to be performed by the user is not displayed in thismodified example as described above, in accordance with the position ofthe first finger FF1 varying after the image IM6 is displayed, the imagesetting section 165 transmits the control signal for operating thetelevision set TV1 to the control device 300.

Although in the embodiments described above, the image determinationsection 168 detects whether or not the image of the operation object orthe determination object is included in the taken image taken by thecamera 61, the method of detecting the operation object or thedetermination object can variously be modified. For example, it is alsopossible for an infrared sensor or an ultrasonic sensor to detect theoperation object or the determination object. Further, it is alsopossible for a radar detector for detecting a radar wave to detect theoperation object or the determination object.

D3. Modified Example 3

Although in the third embodiment described above, the wirelesscommunication section 132 b of the HMD 100 b obtains the positioninformation of the control apparatus and the controller of the controlapparatus as the control devices 300 b from the storage section 420 ofthe server 400, it is also possible that the position information to beobtained is only the position information of either one of the controlapparatus and the controller. For example, in the case in which thecontrol apparatus and the controller are integrated with each other, itis sufficient for the wireless communication section 132 b to obtain theposition information of either one of the control apparatus and thecontroller. Further, it is also possible that the position informationof one of the control apparatus and the controller is obtained via thewireless communication section 132 b, and the position information ofthe other is obtained by the image recognition as in the firstembodiment and the second embodiment.

Although in the third embodiment described above, the image settingsection 165 makes the image of the same corresponding markers areprovided to the control apparatus thus detected and the controller ofthe control apparatus thus detected in the maximum image display area PNas shown in FIGS. 18 and 19, the corresponding markers to be displayedcan variously be modified. For example, it is also possible to make thecontrol apparatus and the controller be displayed in the maximum imagedisplay area PN as the images having the corresponding markers the samein shape, and different in color. Further, the character image such as“CONTROL APPARATUS” or “CONTROLLER” can be displayed in the maximumimage display area PN. Further, the image of the arrow indicating bothof the control apparatus and the controller can be displayed in themaximum image display area PN as the image representing thecorrespondence relationship between the control apparatus and thecontroller. Further, the distances to the control apparatus and thecontroller are measured as in the first and second embodiments, and itis possible for the image setting section 165 to make the images of thecorresponding markers be displayed in the maximum image display area PNwith the sizes of the images varied in accordance with the distancesthus measured.

Although in the third embodiment described above, the external sensor 70detects the acceleration as the change in the position of the externalsensor 70, the physicality value to be detected as the change in theposition can variously be modified. For example, the external sensor 70can also be a gyro sensor for detecting the angular velocity. Further,the external sensor 70 does not need to have a rod-like shape, and canhave such a shape as a ring to be mounted on a finger of the user or awristwatch to be mounted on an arm of the user. Further, the externalsensor 70 can be deformed, and it is also possible to measure thedeformation amount of the external sensor 70 as the change in theposition of the external sensor 70.

Although in the third embodiment described above, the camera 61 imagesthe external sight, and the position identification section 163 correctsthe position information of the control device 300 b using the image ofthe control device 300 b included in the taken image, it is notnecessary required to image the external sight. The position of thecontrol device 300 b can also be identified based only on the positioninformation obtained via the wireless communication section 132 b.

FIG. 21 is an explanatory diagram showing an example of the visual fieldVR to be visually recognized by the user in the case in which positioninformation of the control apparatus has been obtained by iBeacon (aregistered trademark). FIG. 21 shows the visual field VR of the user forvisually recognize the external sight SC including the image displayedin the maximum image display area PN in the case in which the HMDobtains the optical information, which has been transmitted from theinformation transmission section IB attached to the television set TV1as the control apparatus, using the mechanism of iBeacon. In thismodified example, the position identification section 163 obtains theposition information of the television set TV1 not from the server 400,but from the information transmission section IB of the television setTV1. As described above, the position identification section 163 canobtain the position information of the control device 300 b from otherdevice using a variety of methods. As means for the communication, it ispossible to use, for example, power line communication or the like.

FIG. 22 is an explanatory diagram showing the visual field VR to bevisually recognized by the user in the case in which an image IM10 hasbeen displayed in the maximum image display area PN in accordance withthe control instruction performed. In this modified example, the imagesetting section 165 makes the image associated with the controlapparatus MC thus detected be displayed in the maximum image displayarea PN for each of the users identified by the identifying section 161.As shown in FIG. 22, the image IM10, which shows the relationshipbetween the control apparatus MC and a peripheral device related to thecontrol apparatus MC using a block diagram, is displayed in the maximumimage display area PN. As described above, the image to be displayed inthe maximum image display area PN with the control instruction thusexecuted can variously be modified.

Although in the third embodiment described above, the controlinstruction is executed based on the change in the position of theexternal sensor 70, the operation necessary for executing the controlinstruction can variously be modified. For example, it is also possiblefor the image setting section 165 to determine the control instructionto be executed based on the combination of the sound obtained via themicrophone 69 and the change in the position of the external sensor 70detected. In the HMD according to this modified example, since thecontrol instruction is determined based on a plurality of elements,namely the sound and the change in the position of the external sensor70, the user can input a larger number of operations, and thus, theusability for the user is improved.

Although in the embodiments described above, the image setting section165 makes the image associated with the control instruction to beexecuted be displayed in the maximum image display area PN, it is notnecessarily required to make the content associated with the controlinstruction be displayed in the maximum image display area PN as animage. For example, it is possible for the image setting section 165 todisplay the image representing the positions of the operation objectdetected and the determination object in the maximum image display areaPN, but it is not required to display the image related to the controlinstruction to be executed in the maximum image display area PN.

It should be noted that selecting of the object in the appended claimsincludes selecting one from a plurality of selectable objects, andfurther includes selecting one from two alternatives, namely selectingand not selecting, with respect to the one selectable object thusdetected.

D4. Modified Example 4

Although in the embodiments described above, the operation section 135is provided to the control section 10, the configuration of theoperation section 135 can variously be modified. For example, there canbe adopted a configuration in which a user interface as the operationsection 135 is disposed separately from the control section 10. In thiscase, since the operation section 135 is separated from the controlsection 10 provided with the power supply 130 and so on, and cantherefore be miniaturized, and thus, the operability of the user isimproved. Further, by providing a 10-axis sensor for detecting themotion of the operation section to the operation section 135 to therebyperform a variety of operations based on the motion thus detected, theuser can instinctively operate the HMD 100.

For example, it is also possible for the image light generation sectionto have a configuration including an organic EL (organicelectroluminescence) display and an organic EL control section. Further,the image generation section, for example, can also use a LCOS (LiquidCrystal On Silicon; LCoS is a registered trademark), a digitalmicromirror device, and so on instead of the LCD. Further, for example,it is also possible to apply the invention to a laser retinal projectionhead mounted display. In the case of the laser retinal projection type,the maximum image display area PN can be defined as an image area to berecognized by the eyes of the user.

Further, for example, the HMD 100 can also be formed as a head mounteddisplay having a configuration in which each of the optical imagedisplay sections covers only a part of the eye of the user, in otherwords, a configuration in which each of the optical image displaysections does not completely cover the eye of the user. Further, it isalso possible to assume that the HMD 100 is a so-called monocular headmounted display.

FIGS. 23A and 23B are explanatory diagrams each showing an exteriorconfiguration of the HMD 100 according to a modified example. In thecase of the example shown in FIG. 23A, the HMD 100 is different from theHMD 100 shown in FIG. 1 in the point that the image display section 20 bis provided with a right optical image display section 26 b instead ofthe right optical image display section 26, and the point that the leftoptical image display section 28 b is provided instead of the leftoptical image display section 28. The right optical image displaysection 26 b is formed to be smaller than the optical member of theembodiment described above, and is disposed obliquely above the righteye of the user when wearing the HMD 100 b. Similarly, the left opticalimage display section 28 b is formed to be smaller than the opticalmember of the embodiments described above, and is disposed obliquelyabove the left eye of the user when wearing the HMD 100 b. In the caseof the example shown in FIG. 23B, the differences from the HMD 100 shownin FIG. 1 are the point that the image display section 20 c is providedwith a right optical image display section 26 c instead of the rightoptical image display section 26, and the point that the left opticalimage display section 28 c is provided instead of the left optical imagedisplay section 28. The right optical image display section 26 c isformed to be smaller than the optical member of the embodimentsdescribed above, and is disposed obliquely below the right eye of theuser when wearing the head mounted display. The left optical imagedisplay section 28 c is formed to be smaller than the optical member ofthe embodiments described above, and is disposed obliquely below theleft eye of the user when wearing the head mounted display. As describedabove, it is sufficient for each of the optical image display sectionsto be disposed in the vicinity of the eye of the user. Further, the sizeof the optical member forming the optical image display sections isdetermined arbitrarily, and it is possible to realize the HMD 100 havinga configuration in which the optical image display sections each coveronly a part of the eye of the user, in other words, the configuration inwhich the optical image display sections each do not completely coverthe eye of the user.

Further, as the earphones, an ear hook type or a headband type can beadopted, or the earphones can be eliminated. Further, it is alsopossible to adopt a configuration as the head-mounted display installedin a mobile object such as a vehicle or a plane. Further, it is alsopossible to adopt a configuration as the head mounted displayincorporated in a body protector such as a helmet.

The configurations of the HMD 100 in the embodiments described above areillustrative only, and can variously be modified. For example, it isalso possible to eliminate one of the direction keys 16 and the trackpad 14 provided to the control section 10, or to provide anotheroperating interface such as an operating stick in addition to or insteadof the direction keys 16 and the track pad 14. Further, it is alsopossible to assume that the control section 10 has a configuration inwhich an input device such as a keyboard or a mouse is connected to thecontrol section 10, and receives an input from the keyboard or themouse.

Further, it is also possible to adopt an image display section ofanother system such as an image display section to be worn like a hat asthe image display section instead of the image display section 20 to beworn like a pair of glasses. Further, the earphones 32, 34 canarbitrarily be omitted.

Further, in the embodiments described above, it is also possible toassume that the HMD 100 guides the image light beams representing thesame image to the right and left eyes of the user to thereby make theuser visually recognize a two-dimensional image, or to assume that thehead-mount type display device 100 guides the image light beamsrepresenting respective images different from each other to the rightand left eyes of the user to thereby make the user visually recognize athree-dimensional image.

Further, in the embodiments described above, it is also possible toreplace a part of the configuration realized by hardware with software,or by contraries, to replace a part of the configuration realized bysoftware with hardware. For example, although in the embodimentsdescribed above, it is assumed that the image processing section 160 andthe sound processing section 170 are realized by the CPU 140 retrievingand then executing the computer program, it is also possible to assumethat these functional sections are realized by hardware circuits. Forexample, some of the constituents provided to the control section 10according to the embodiments described above can also be configuredusing an application specific integrated circuit (ASIC) designed torealize the function of the invention.

Further, in the case in which a part or the whole of the function of theinvention is realized by software, the software (the computer program)can be provided in a form of being stored in a computer-readablerecording medium. In the invention, the “computer-readable recordingmedium” is not limited to a portable recording medium such as a flexibledisk or a CD-ROM, but includes an internal storage device in thecomputer such as a variety of types of RAM or ROM, and an externalstorage device fixed to the computer such as a hard disk drive.

Further, although in the embodiment described above, the control section10 and the image display section 20 are formed as the separateconstituents as shown in FIGS. 1 and 2, the configuration of the controlsection 10 and the image display section 20 is not limited to thisconfiguration, but can variously be modified. For example, theconfiguration formed in the control section 10 can be formed inside theimage display section 20 in whole or in part. Further, it is alsopossible to adopt a configuration in which the power supply 130 in theembodiment described above is formed alone, and can be replaced, or theconfiguration provided to the control section 10 can redundantly beprovided to the image display section 20. For example, the CPU 140 shownin FIG. 2 can also be provided to both of the control section 10 and theimage display section 20, and it is also possible to adopt aconfiguration in which the functions respectively performed by the CPU140 provided to the control section 10 and the CPU provided to the imagedisplay section 20 are separated from each other.

Further, it is also possible to adopt a configuration of a wearablecomputer in which the control section 10 and the image display section20 are integrated with each other, and can be attached to the clothes ofthe user.

The invention is not limited to the embodiments and the modifiedexamples described above, but can be realized with a variety ofconfigurations within the scope or the spirit of the invention. Forexample, the technical features in the embodiments and the modifiedexamples corresponding to the technical features in the aspectsdescribed in SUMMARY section can arbitrarily be replaced or combined inorder to solve all or a part of the problems described above, or inorder to achieve all or a part of the advantages described above.Further, the technical feature can arbitrarily be eliminated unlessdescribed in the specification as an essential element.

The entire disclosure of Japanese Patent Application Nos. 2014-094613,filed May 1, 2014 and 2015-024971, filed Feb. 12, 2015 are expresslyincorporated by reference herein.

What is claimed is:
 1. A head-mounted display device comprising: adisplay configured to display virtual imagery, and at least one of:capable of transmitting an external sight through the display, andcapable of transmitting the external sight via video transmission; andat least one processor or circuit configured to: control a camera toacquire images, and while the camera is acquiring images: detect, byimage processing, whether or not the acquired images include image datathat is the same as image data of an operation object stored in amemory; and when the acquired images include image data that is the sameas the image data of the stored operation object: determine whether theoperation object is located within a predetermined distance range fromthe display, and if the operation object is located within thepredetermined distance range: control the display to display a virtualimage associated with the operation object; detect, by image processing,a gesture associated with the operation object; and when the gestureassociated with the operation object has been detected, control thedisplay to display a predetermined image associated with the operationobject.
 2. The head-mounted display device according to claim 1, whereinthe predetermined image is an image for prompting a user of thehead-mounted display device to perform a check action of a numericalvalue indicated by the operation object.
 3. The head-mounted displaydevice according to claim 2, wherein the processor or circuit is furtherconfigured to: control the display to display a gesture instructionassociated in advance with the operation object.
 4. The head-mounteddisplay device according to claim 3, wherein the processor or circuit isfurther configured to: control the display to display a virtual imagerepresenting the change in the position of the specific object necessaryfor executing the control of the operation object as the gestureinstruction.
 5. A control system comprising: the head-mounted displaydevice according to claim 4; and a server processor configured to:identify the operation object; transmit information about the identifiedoperation object; wherein the processor or circuit of the head-mounteddisplay device is further configured to: receive the transmittedinformation about the identified operation object, and transmitinformation about the control of the operation object so that the serverprocessor can execute control based on the transmitted information aboutthe control of the operation object.
 6. A control system comprising: thehead-mounted display device according to claim 3; and a server processorconfigured to: identify the operation object; and transmit informationabout the identified operation object, wherein the processor or circuitof the head-mounted display device is further configured to: receive thetransmitted information about the identified operation object, andtransmit information about control of the operation object so that theserver processor can execute control based on the transmittedinformation about the control of the operation object.
 7. Thehead-mounted display device according to claim 1, wherein the processoror circuit is further configured to: obtain a position of a specificobject; detect the gesture associated with the operation object based onthe obtained position of the specific object; and when the gestureassociated with the operation object has been detected, execute acontrol instruction associated with the operation object.
 8. Thehead-mounted display device according to claim 7, wherein the processoror circuit is further configured to: control the display to display avirtual image representing a content of the control of the operationobject to be executed.
 9. The head-mounted display device according toclaim 7, wherein the processor or circuit is further configured to: in acase in which the position of the specific object has changed and thespecific object and operation object have positions that overlap eachother, determine that the gesture associated with the operation objecthas been detected.
 10. The head-mounted display device according toclaim 7, further comprising: a sound acquisition device adapted toobtain an external sound, wherein the processor or circuit is furtherconfigured to: determine the control of the operation object based on acombination of the change in the position of the specific objectidentified and the sound obtained by the sound acquisition device.
 11. Acontrol system comprising: the head-mounted display device according toclaim 7; and a server processor configured to: identify the operationobject; and transmit information about the identified operation object;wherein the processor or circuit of the head-mounted display device isfurther configured to: receive the transmitted information about theidentified operation object, and transmit information about control ofthe operation object so that the server processor can execute controlbased on the transmitted information about the control of the operationobject.
 12. The head-mounted display device according to claim 1,further comprising: a sensor configured to measure a distance betweenthe operation object and the display, wherein the processor or circuitis further configured to: set the operation object as a selectableobject, in a case that the measured distance between the operationobject and the display is equal to or shorter than a threshold value.13. The head-mounted display device according to claim 1, wherein theprocessor or circuit is further configured to obtain positioninformation of the operation object from another device.
 14. Thehead-mounted display device according to claim 13, wherein the processoror circuit is further configured to: when the detected operation objectfails to be included in the external sight, control the display todisplay a positional relationship between the display and the detectedoperation object.
 15. The head-mounted display device according to claim1, wherein the processor or circuit is further configured to: detect anacceleration of a specific object, and identify a change in the positionof the specific object based on the detected acceleration of thespecific object.
 16. The head-mounted display device according to claim15, further comprising: a sound acquisition device configured to obtainan external sound, wherein the processor or circuit is furtherconfigured to: determine the control of the operation object, based onthe obtained external sound and the identified change in the position ofthe specific object.
 17. The head-mounted display device according toclaim 1, wherein the processor or circuit is further configured to:determine whether a control device associated with the operation objectis within the external sight, and in a case in which the operationobject and the control device are both within the external sight,control the display to display a virtual image indicating that theoperation object and the control device are in a correspondencerelationship.
 18. The head-mounted display device according to claim 17,wherein the processor or circuit is further configured to: control thedisplay to display virtual images having identical shapes and identicalcolors as the virtual image indicating that the operation object and thecontrol device are in the correspondence relationship.
 19. Thehead-mounted display device according to claim 1, wherein the processoror circuit is further configured to: identify an attribute of a user ofthe head-mounted display device, and control the display to display atleast one of a virtual image associated with the operation object andthe specific check image as the virtual image so as to correspond to theattribute identified.
 20. The head-mounted display device according toclaim 1, wherein the processor or circuit is further configured to:determine whether a part of a control device associated with theoperation object is within the external sight; and in a case in which nopart of the control device is included in the imagining range of thecamera: generate bird's-eye view imagery, which shows a positionalrelationship between the user and the control device; and control thedisplay to display the bird's-eye view imagery.
 21. The head-mounteddisplay device according to claim 1, further comprising: a sensorconfigured to measure a distance between the operation object and thedisplay, wherein the processor or circuit is further configured to:determine whether a part of a control device associated with theoperation object is within the external sight; and when a part of thecontrol device associated with the operation object is within theexternal sight and a threshold change in position of the external sensorhas been detected, execute a control instruction of the control devicecorresponding to the threshold change in the position of the externalsensor.
 22. The head-mounted display device according to claim 1,wherein the processor or circuit is further configured to: measure thedistance between the operation object and the display using at least oneof: a TOF sensor and an infrared LED.
 23. The head-mounted displaydevice according to claim 1, wherein the predetermined image includes aplurality of icons representing contents to be controlled if theposition of the finger overlaps one of the icons.
 24. A method ofcontrolling a head-mounted display device including a display configuredto display virtual imagery, and at least one of: capable of transmittingan external sight through the display, and capable of transmitting theexternal sight via video transmission, the method comprising:controlling a camera to acquire images, and while the camera isacquiring images: detecting, by image processing, whether or not theacquired images include image data that is the same as image data of anoperation object stored in a memory; and when the acquired imagesinclude image data that is the same as the image data of the storedoperation object: determining whether the operation object is locatedwithin a predetermined distance range from the display, and if theoperation object is located within the predetermined distance range:controlling the display to display a virtual image associated with theoperation object; detecting, by image processing, a gesture associatedwith the operation object; and when the gesture associated with theoperation object has been detected, controlling the display to display apredetermined image associated with the operation object.
 25. Anon-transitory computer readable medium storing computer-executableprogram instructions used with a head-mounted display device including adisplay configured to display virtual imagery and at least one of:capable of transmitting an external sight through the display, andcapable of transmitting the external sight via video transmission, thecomputer-executable program instructions, when executed by at least oneprocessor of the head-mounted display device, cause the at least oneprocessor to: control a camera to acquire images, and while the camerais acquiring images: detect, by image processing, whether or not theacquired images include image data that is the same as image data of anoperation object stored in a memory; and when the acquired imagesinclude image data that is the same as the image data of the storedoperation object: determine whether the operation object is locatedwithin a predetermined distance range from the display, and if theoperation object is located within the predetermined distance range:control the display to display a virtual image associated with theoperation object; detect, by image processing, a gesture associated withthe operation object; and when the gesture associated with the operationobject has been detected, control the display to display a predeterminedimage associated with the operation object.
 26. A head-mounted displaydevice comprising: a display configured to display virtual imagery, andat least one of: capable of transmitting an external sight through thedisplay, and capable of transmitting the external sight via videotransmission; and at least one processor or circuit configured to:control a camera to acquire images, and while the camera is acquiringimages: detect, by image processing, whether or not the acquired imagesinclude image data that is the same as image data of two or moreoperation objects stored in a memory; and when the acquired imagesinclude image data that is the same as the image data of the two or morestored operation objects: control the display to display a virtual imageassociated with the two or more operation objects; detect, by imageprocessing, a gesture associated with at least one of the at least twooperation objects; and when the gesture associated with the at least oneoperation object has been detected, control the display to display apredetermined image associated with the at least one operation object.27. A method of controlling a head-mounted display device including adisplay configured to display virtual imagery, and at least one of:capable of transmitting an external sight through the display, andcapable of transmitting the external sight via video transmission, themethod comprising: controlling a camera to acquire images, and while thecamera is acquiring images: detecting, by image processing, whether ornot the acquired images include image data that is the same as imagedata of two or more operation objects stored in a memory; and when theacquired images include image data that is the same as the image data ofthe two or more stored operation objects: controlling the display todisplay a virtual image associated with the two or more operationobjects; detecting, by image processing, a gesture associated with atleast one of the at least two operation objects; and when the gestureassociated with the at least one operation object has been detected,controlling the display to display a predetermined image associated withthe at least one operation object.
 28. A non-transitory computerreadable medium storing computer-executable program instructions usedwith a head-mounted display device including a display configured todisplay virtual imagery and at least one of: capable of transmitting anexternal sight through the display, and capable of transmitting theexternal sight via video transmission, the computer-executable programinstructions, when executed by at least one processor of thehead-mounted display device, cause the at least one processor to:control a camera to acquire images, and while the camera is acquiringimages: detect, by image processing, whether or not the acquired imagesinclude image data that is the same as image data of two or moreoperation objects stored in a memory; and when the acquired imagesinclude image data that is the same as the image data of the two or morestored operation objects: control the display to display a virtual imageassociated with the two or more operation objects; detect, by imageprocessing, a gesture associated with at least one of the at least twooperation objects; and when the gesture associated with the at least oneoperation object has been detected, control the display to display apredetermined image associated with the at least one operation object.